Computer User Interface System and Methods

ABSTRACT

Systems and methods may provide user control of a computer system via one or more sensors. Also, systems and methods may provide automated response of a computer system to information acquired via one or more sensors. The sensor(s) may be configured to measure distance, depth proximity and/or presence. In particular, the sensor(s) may be configured to measure a relative location, distance, presence, movements and/or gestures of one or more users of the computer system. Thus, the systems and methods may provide a computer user interface based on measurements of distance, depth, proximity, presence and/or movements by one or more sensors. For example, various contexts and/or operations of the computer system, at the operating system level and/or the application level, may be controlled, automatically and/or at a user&#39;s direction, based on information acquired by the sensor(s).

This application is a continuation of U.S. patent application Ser. No.12/238,601, filed Sep. 26, 2008, which is hereby incorporated byreference herein in its entirety. This application claims the benefit ofand claims priority to U.S. patent application Ser. No. 12/238,601,filed Sep. 26, 2008.

TECHNICAL FIELD

This invention relates generally to computer user interface systems andmethods, and more specifically to computer user interface systems andmethods employing distance, depth and/or proximity sensors.

BACKGROUND

A wide variety of computer devices are known, including, but not limitedto, personal computers, laptop computers, personal digital assistants(PDAs), and the like. Such computer devices are typically configuredwith a user interface that allows a user to input information, such ascommands, data, control signals, and the like, via user interfacedevices, such as a keyboard, a mouse, a trackball, a stylus, a touchscreen, and the like.

Other devices, such as kiosks, automated teller machines (ATMs), and thelike, may include a processor or otherwise be interfaced with aprocessor. Accordingly, such devices may essentially include a computerand thus may include a similar user interface as employed for computersystems.

SUMMARY

Various embodiments described herein are directed to computer userinterface systems and methods that provide user input to a computerbased on data obtained from at least one distance, depth and/orproximity sensor associated with the computer. In particular, variousembodiments involve a plurality of distance, depth and/or proximitysensors associated with the computer.

Various embodiments contemplate a computer user interlace systems thatallow a computer to be controlled or otherwise alter its operation basedon information detected by such sensors. In some embodiment, anoperating system of the computer may be controlled based on suchinformation. Alternatively or additionally, an active application may becontrolled based on such information.

In particular, the information detected by such sensors may definevarious user contexts. In other words, the sensor(s) may detect a userparameter relative to the computer such that different user contexts maybe determined. For example, a user presence context may be determined tobe presence of a user, absence of a user or presence of multiple userswithin a vicinity of the computer. Based on the determined user presencecontext, the computer may be placed in a particular mode of operationand/or may alter an appearance of information displayed by the computer.

Various other user contexts may be determined based on information oruser parameters detected by one or more sensors. A user proximitycontext may be determined, for example, in terms of a location of a userrelative to the compute or a distance of a user from the computer. Insuch embodiments, an appearance of information displayed by the computermay be altered or otherwise controlled based on the user proximitycontext. For example, a size and/or a content of the informationdisplayed by the computer may be altered or controlled.

Also, in some embodiments, a user gesture context may be determinedbased on information or user parameters detected by one or more sensors.In such embodiments, one or more operations of the computer may beperformed based on the determined user gesture context. For example,such operations may include, but are not limited to, scrolling,selecting, zooming, or the like. In various embodiments, such operationsmay be applied to an active application of the computer. In general,user gesture contexts may be employed to allow a user to operate thecomputer remotely via the one or more sensors, without a remotecontroller or other auxiliary user manipulated input device.

Thus, various embodiments contemplate a sensor-based computer userinterface system. Other embodiments contemplate a method userinteraction with a computer system via one or more sensors. Stillfurther embodiments contemplate a computer readable storage mediumincluding stored instructions that, when executed by a computer, causethe computer to perform any of the various methods described hereinand/or any of the functions of the systems disclosed herein.

These and other embodiments and features will be apparent to those ofordinary skill in the art upon reading this disclosure in its entirety,along with the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic representation of a monitor of a computer systemincluding a sensor-based user interface.

FIG. 2 is a schematic representation of a keyboard of a computer systemincluding a sensor-based user interface.

FIG. 3 is a schematic block diagram illustrating an example of a sensorbased user interface for a computer system.

FIG. 4 is a block diagram illustrating a more detailed example of thesensor based user interface system shown in FIG. 3.

FIG. 5 is a flowchart illustrating an example of a method of userinteraction with a computer system via one or more sensors.

FIGS. 6A and 6B are schematic representations of information that may bedisplayed based on the method of user interaction illustrated in theflowchart of FIG. 5.

FIGS. 7A and 7B are schematic representations of other information thatmay be displayed based on the method of user interaction illustrated inthe flowchart of FIG. 5.

FIG. 8 is a flowchart illustrating another example of a method of userinteraction with a computer system via one or more sensors.

DETAILED DESCRIPTION OF TUE INVENTION

Computer systems and methods disclosed herein may provide an entirelynew category of user interface systems as compared to conventional userinterfaces. Various embodiments contemplate using one or more sensors asa means for a user to interact with a computer system, either separatelyor in conjunction with conventional peripheral user input devices, suchas mice, keyboards, and the like.

In some embodiments, such an approach may allow a user to interact witha computer remotely, that is, at a distance outside of physical reach,without using a remote controller or other user manipulated inputdevice. In some embodiments, such an approach may allow a computersystem to automatically perform various operations in response tochanges in user contexts.

As used herein, a user context should be understood as defining how auser relates to a computer system in terms of relative location,distance or proximity, presence or absence, and/or movements orgestures. A user context should be understood as defining suchrelationships to the computer for multi-user settings as well. It shouldalso be understood that additional information may form part of the usercontext, including but not limited to, information regarding theactivity of the user, the identity of the user, the time and/or date,the location of the computer system, the environment of the computersystem/user(s), active application(s), operating system variables, andthe like. In other words, any relevant information available to thecomputer system regarding the user(s) and his environment that affectsthe user's(s') interaction with the computer system may be included, asappropriate or desired, to define one or more user contexts. Asdescribed herein, sensor data may provide certain information regardingthe user(s), while other information may be provided by other aspects ofthe computer system, such as user account settings and/or preferences,open applications and/or windows, current/active application,application history, time, date, location, IP address, and the like.

As used herein, a user parameter should be understood as including anydetectable parameter associated with a user. This may include a user'sphysical being, for example, based on which presence, depth, distance,proximity, movements, gestures, and the like may be detected. In somecases, this may also include objects that a user may carry or keep onhis person, for example, based on which presence, depth, distance,proximity and/or changes in such values may be detected. For example, auser's wireless communication device (e.g., cell phone, electronicidentification card or tag, and the like) may be detected, for example,via wireless communication protocols, such as Bluetooth® or RFID (radiofrequency identification).

The systems and methods described herein may provide a sensor-based userinterface and/or ways for a user to interact with a computer system viaone or more sensors. The sensor or sensors may be of any suitable type,either currently known or hereafter developed, that is configured todetect one or more of depth, distance, proximity, presence, or the like.For example, approaches such as near field radio frequency (RF),ultrasonic, infrared (IR), antenna diversity, or the like may beemployed. This lists is not intended to be exhaustive, and it should beunderstood that other sensors may be employed as well, including, butagain not limited to, visible light sensors, ambient light sensors,mechanical vibration sensors, it should be understood that the termsensors as used herein is intended to include systems of emitters andsensors, for example, where the emitters “paint” a field of interest andthe sensors detect resulting reflections of objects within the field. Itshould also be understood that the number of sensors as well as thetype, the sensitivity, the range, the field of detection (angle ofincidence), and the like, may be determined, for example, on the userinteraction to be detected and/or the computer system control/responseto be implemented. Further, it should be understood that the sensorscontemplated herein are relatively “simple” sensors that do notgenerally require intensive data processing and may provide suitable“reaction time” to changing parameters that indicate changes in usercontexts. This does not mean that more complex, processing-intensivesystems may not be employed, but rather that such more complex systemsmay be invoked based on the relatively simple and reactive sensorsystems contemplated herein.

In some embodiments, a plurality of sensors may be employed. Based on aknown distribution of sensors, such as a linear or two-dimensional arrayof sensors, various algorithms may be employed to calculate relativestates of the sensors and/or changes in the states of the sensors and tomap or interpret the states/changes to determine user contexts and/orchanges in user contexts. In general, sensor systems employing digitalsignal processing techniques and state machine logic may be employed toimplement the various features described herein.

For gesture extraction, it should be understood that the sensor systemand/or context engine may be calibrated and trained to learn a specificset of gestures, for example, from a specific user. Such a process maybe adaptive, for example, beginning with a relatively small set ofgestures that are more readily detectable orrecognizable/distinguishable, and building a user-dependent gesturedatabase with more complex gestures through training and/or use.

Gestures may have time and/or sequence dependence, for example, suchthat a specific sequence of detected user movements may be interpretedas a gesture. Gestures may also be distance dependent, for example, suchthat gestures include a distance component. In general, it should beunderstood that the gesture contexts may be dependent on other usercontexts, as discussed herein. The user contexts and changes in usercontexts, whether gesture contexts or not, may have correspondingcontrol instructions or commands that are provided to the computersystem to effect user control and/or interaction with the computersystem.

In some embodiments, the sensor(s) may be configured to measure values.Alternatively or additionally, the sensor(s) may be employed in a binarymanner, either on or off. In general, the sensors contemplated hereinmay measure analog values that may be converted to digital signals andprocessed by a digital signal processor (DSP), which may be implementedin hardware and/or software. The processed digital signals may beprovided to threshold detection logic, for example, to generate binaryinformation about the user context (such as user present or absent), maybe provided to multi-threshold logic, for example, to determineinformation having more than two states (such as, usernear/intermediate/far or user left/center/right) and/or may be used as adigital measure (such as user distance or relative location).

As will be further understood from the following description, the usercontexts that may be determined based on detected user parameter(s) mayinclude, for example, a user presence context, a user proximity contextand a user gesture context. While these contexts are describedseparately, it should be understood that one or more of such contextsmay be combined to define a user context of interest. As discussedabove, other information may form part of the user context(s) as well.

In the sensor-based computer user interface systems and methodsdescribed herein, the control of and/or response by the computer systemmay be at an operating system level or at an application level. Forexample, on the operating system level, the operating system mayincrease screen brightness if the user is determined to be far from thecomputer display. The operating system may transfer control of adisplayed pointer from a mouse, for example, when the user is near, togesture detection, for example, when the user is far. In someembodiments, an active application may be controlled or otherwiserespond to the information detected by the sensor(s).

Turning now to particular embodiments that provide examples of howsensor-based computer user interface systems and methods may beimplemented, a schematic representation of a monitor 100 of a computersystem including a sensor-based user interface is shown in FIG. 1. Themonitor 100 may include a screen 102 that is surrounded by a frame 104.

The frame 104 of the monitor 100 may provide a suitable location forsensors 110. As a user or users may typically be positioned in a mannerto view the screen 102, the sensors 110 may be mounted to the frame 104or otherwise incorporated into the monitor 100 in a manner that will beeffective for detecting the user(s) and/or user parameters.

Although the sensors 110 are shown as being located at top, bottom, leftand right positions, it should be understood that other arrangements arepossible and may be suitable for a given application. For example, anarrangement of a plurality of sensors 110 may be suitable for detectinguser parameters and/or changes in three dimensions. Also, it should beunderstood that a single sensor 110 may be employed for someapplications. For example, a single proximity sensor may be employed todetect user presence/absence and/or user near/far.

FIG. 2 is a schematic representation of a keyboard 200 of a computersystem including a sensor-based user interface. The keyboard 200 mayinclude keys 202 that are disposed within a frame 204. The frame 204 ofthe keyboard 200 may provide a suitable location for sensors 210. As auser or users may typically be positioned in a manner to use thekeyboard 200, the sensors 210 may be mounted to the frame 204 orotherwise incorporated into the keyboard 200 in manner that will beeffective for detecting the user(s) and/or user parameters.

It should be understood that the sensor arrangement illustrated in FIG.2 may be employed in addition, or as an alternative, to the sensorarrangement illustrated in FIG. 1. Further, it should be understood thatother sensor arrangements may be employed.

In general, the sensor(s) need only be configured to detect a userparameter and/or changes in a user parameter. For example, sensors thatare not attached or incorporated into a structure of the computer systemmay be employed. For example, it may be sufficient to place thesensor(s) at a desired location of use to allow a user to control and/orinteract with the computer system from the desired location. In, someembodiments in which the sensor or sensors is/are not attached orincorporated into a structure of the computer system, a mechanism may beprovided to detect or, determine a location of, the sensor(s) relativeto the computer system or a part thereof. In other words, wired orwireless communication between the sensors and the rest of the systemwould be needed, and, at least for some user contexts, the relativelocation(s) of the sensor(s) to the computer system and/or each othermay be needed.

FIG. 3 is a schematic block diagram illustrating an example of asensor-based user interface system 300 for a computer system 302. Asshown, the computer system 302 may be considered to form part of thesensor-based user interface 300 in some embodiments. For example, aprocessor of the computer system 302 may be employed to controloperations of the various components of the sensor-based user interfacesystem 300. However, it should be understood that the sensor-based userinterface system 300 may be defined separately from the computer system302 with which it is configured to interact, for example, by including aseparate processor. As such, it should be understood that the blockdiagram of FIG. 3 is intended only to be illustrative. Also, it shouldbe understood that details of the computer system 302 not specificallydiscussed herein may be implemented in any suitable manner, such asconventional and well known in the art.

As discussed above, the sensor-based user interface, system 300 mayinclude one or more sensors 310. Operation of the sensors 310 may besupported by a sensor processing or data acquisition module 320. Thesensor processing/data acquisition module 320 may be implemented assoftware and/or hardware, as appropriate or desired. For example, thesensor processing/data acquisition module 320 may include analog todigital conversion, digital signal processing (such as filtering,thresholds and the like), such as illustrated in FIG. 4, discussedbelow.

Data or information acquired and/or processed from the sensor(s) 310 bythe sensor processing/data acquisition module 320 may be provided to auser context engine 330. The user context engine 330 may be implementedas software and/or hardware, as appropriate or desired. For example, theuser context engine 330 may be implemented as a state machine withvarious levels of complexity, may be based on or incorporate neuralnetworks, fuzzy logic, statistical model building and reasoning based onBayesian classifications, or the like. In general, the user contextengine 330 may be configured to interpret the information from thesensor(s) 310 to determine a corresponding user context and/or changesin the user context.

In particular, the user context engine 330 may employ one or morealgorithms to determine user contexts and/or changes in user contexts.The user context engine 330 may access a user context database 340 todetermine whether the result(s) of the algorithm(s) correspond todefined user contexts and/or changes in user contexts. Once a usercontext or change in user context is determined, the user context engine330 may access a corresponding control instruction, for example, storedin the user context database 340, and provide the corresponding controlinstruction to the computer system 302 to effect a desired control ofand/or response by the computer system 302.

Alternatively, the user context engine 330 may provide the determineduser context and/or change in user context to the computer system 302,which may be configured to implement a corresponding controlinstruction, example, the computer system 302 may be configured to mapuser contexts and/or changes in user contexts to corresponding controlinstructions.

In addition to the information provided by the sensor processing/dataacquisition module 320, the user context engine 330 may be configured toreceive information from the computer system 302 regarding its operatingstate. For example, information regarding the particular operationalstate of, a particular active application on, and/or particularinformation for display by the computer system 302 may be used by theuser context engine 330 to interpret the information received from thesensor(s) to determine the user contexts and/or the changes in usercontexts and/or to map the user contexts and/or the changes in usercontexts to a particular set of control instructions.

The user context engine 330 may also be configured to “learn” or to be“trained” to recognize various user contexts and/or changes in usercontexts, for example, by executing a training algorithm with a userproviding various inputs via the sensor(s) 310. In such a manner, theuser context database 340 may be populated or “educated.” As appropriateor desired, the user context database 340 may be populated for specificusers so that the sensor-based user interface is tailored, for example,to the characteristics and/or mannerisms of the particular use to betterdefine the user parameters to be detected by the sensor(s) 310. Anysuitable learning approach may be employed, particularly adaptivelearning approaches. In general, the approach should allow the system tobe calibrated to decrease and/or minimize false detection and increaseand/or maximize positive recognition of gestures.

FIG. 4 is a block diagram illustrating a more detailed example of thesensor-based user interface system 300 for the computer system 302 shownin FIG. 3. In FIG. 4, the computer 302 is represented by the operatingsystem (OS), but it should be understood that one or more applicationsand/or hardware may be employed to interact with the sensor-based userinterface system 300. The operating system of the computer 302 may beemployed to control operations of the various components of thesensor-based user interface system 300. However, this should not beconsidered to exclude the possibility of the various components of thesensor-based user interface system 300 including one or more separateprocessors, as appropriate or desired, which may operate independentlyor at the direction of the computer 302.

As discussed above, the sensor-based user interface system 300 mayinclude a plurality of sensors 310 ₁, 310 ₂ . . . 310 _(N). Asillustrated in FIG. 4, the sensors 310 ₁, 310 ₂ . . . 310 _(N) may eachrepresent one or more sensors of a particular type (such as configuredto detect a particular user parameter) or of an associated set ofsensors designed to cooperate in some manner. As such each of thesensors 310 ₁, 310 ₂ . . . 310 _(N) may communicate with a respectivesensor processing module 320 ₁, 320 ₂ . . . 320 _(N). Alternatively, asappropriate or desired, a single sensor processing module may beemployed for multiple ones or all of the sensors 310 ₁, 310 ₂ . . . 310_(N). Thus, it should be understood that separate sensor processingmodules and/or circuitry of a shared processing module may be tailoredto the particular sensor(s) communicating therewith.

For example, the sensors 310 ₁, 310 ₂ . . . 310 _(N) may be analogsensors such that the respective sensor processing modules 320 ₁, 320 ₂. . . 320 _(N) include an analog-to-digital (ND) converter 322. Ofcourse, if the corresponding sensor provides data in digital form.Digital data from the analog-to-digital converter 322 may be provided toeither a slow filter 324 or a fast filter 326, depending on reactiontime, sensitivity or the like desired for the particular user parameterand/or user context of concern. As discussed above, various approachesto threshold settings may be employed, again as appropriate or desiredfor the particular user parameter and/or user context of concern. Forexample, an adaptive threshold circuit or module 327 and/or amulti-level detection logic circuit or module 328 may be employed.

Processed sensor information, such as presence, distance, and the like,as discussed herein, may be provided from the sensor processing modules320 ₁, 320 ₂ . . . 320 _(N) to the user context engine 330. As discussedabove, the user context engine 330 may employ a state machine,illustrated by the user context state module 332 a, the next state logic332 b and the data flow lines, including current state and useroperating system variables, for example, as input to the next statelogic 322 b to determine the updating of the user state context 322 a tothe next state based on the processed sensor information provided.

Also, the user context engine 330 may include gesture detection logic324 that may implement interpretation of the processed sensorinformation to determine whether such information corresponds to variousgesture commands, for example, stored in a database of gesture commandsas discussed above. The gesture detection logic 324 may base itsinterpretation of the processed sensor information based on changes inuser context state and/or other information, such as user OS variablesas shown, provided from the computer system 302.

As will be understood from the foregoing, various methods may beenvisioned for user interaction with a computer system via asensor-based user interface. As such, the following description ofmethods should be understood to be for the sake of illustration, and notlimiting.

FIG. 5 is a flowchart illustrating one example of a method of userinteraction with a computer system via one or more sensors. Control maybegin at S400, and may proceed to S410 where the sensor-based userinterface is enabled. As discussed above, it should be understood thatthe sensor-based user interface may be used in conjunction withconventional user interface devices. Thus, enabling of the sensor-baseduser interface may be via a mouse, a keyboard, or the like, or may beautomatic upon startup of the computer system, its operating system, adedicated application and/or an application that is compatible with thesensor-based user interface.

In S420, an initial user parameter may be detected. As discussed above,the detected user parameter may be the user's body or a part thereof, oran object associated with the user. Based on the detected initial userparameter, an initial user context may be determined AT s430. Forexample, a user presence context may be determined to be that a user ora specific user is present in a certain vicinity of the computer system.

Based on the determined initial user context, an operating system and/oran active application of the computer system may be controlled at S440.For example, when the user presence context is that a user is in thevicinity of the computer system, the computer system may be placed in ause mode—that is, an operational mode in which the computer is runningnormally.

At S450, a determination may be made as to whether or not a change inthe user parameter or user context is detected by the sensor(s). Thismay involve evaluating information received from the sensor(s) todetermine if state(s) of the sensor(s) has/have changed and/or whethersuch change corresponds to a change in the user context. For example, auser may change location relative to the computer system, which maychange the user parameter. However, the change in the user parameter mayor may not be sufficient to cause a change in the user context. However,the user may change his location relative to the computer significantlyenough to result in a change in the user context.

If so, a new user context may be determined at S460. For example, theuser may have changed his location relative to the computer system so asto no longer be within the certain vicinity of the computer system.Based on the new user context, the operating system and/or the activeapplication of the computer system may be controlled at S470. Forexample, when the new user presence context is that a user is not in thecertain vicinity of the computer system, the computer system may beplaced in a non-use mode—that is, an operational mode in which thecomputer is running less active, such as a sleep mode or a hibernatemode. Alternatively, the computer system may automatically logout theuser so that another person cannot access the computer via that user'slogin.

It should be understood that such actions by the computer system may beimplemented only after a predetermined period of time has elapsed afterthe change in user parameter is detected and/or the change in usercontext is determined, as appropriate or desired. For example, a usermay be allowed to leave the certain vicinity for a certain amount oftime without being logged out or having the computer enter a non-usemode.

Control may return to S450 for detection of a further change in userparameter and/or user context. If no change is detected, for example,within a predetermined amount of time, control may continue to S480where a determination may be made as to whether the sensor-based userinterface is disabled. If not, control may return to S440 to continuecontrol of the computer system based on the current context. If so,control may proceed to S490 and end. It should be understood, however,that ending the sensor-based user interface operation may be implementedat any time, for example, based on other user input.

While the flowchart in FIG. 5 is described above in terms of determininga user presence context, it should be understood that other usercontexts mat be determined as well. For example, a user proximitycontext may be determined. This may be similar to the user presencecontext described above, having a certain proximity of the user to thecomputer system defining different user proximity contexts, such as nearand far. Alternatively, user proximity contexts may be defined forvarious discrete distances of the user from the computer system, or evensubstantially continuously varying based on measured distance, asappropriate or desired.

One example of how a computer system may respond or be controlled basedon a change in a user proximity context is illustrated by FIGS. 6A and6B. FIG. 6A shows a computer display 500 including relatively densevisual media 502 with various icons 504 for user interaction, forexample, via a mouse (not shown). The computer display of FIG. 6A may beprovided by the computer system when the user proximity context is thatthe user is relatively near the computer system (or the monitor thereof,for example).

When the user proximity context changes by the user moving to a locationrelatively far from the computer system for the monitor thereof, forexample), the relatively dense visual media 502 may be filtered and/orenlarged to provide information that is usable/viewable from theincreased distance. As shown in FIG. 6B, for example, such a change inuser proximity context may result in a computer display 500 includingfiltered visual media 502, which is an enlarged portion of therelatively dense visual media 502 shown in FIG. 6A. The icons 504 shownin the display 500 of FIG. 6A may not be included, for example, becausethe user may not be within reach of the mouse or other input device forusing such icons 504 in the changed user proximity context. In someembodiments, control of the computer system may be transferred betweenthe mouse or other input device and enabled gesture control.

Another example of how a computer system may respond or be controlledbased on a change in a user proximity context is illustrated by FIGS. 7Aand 7B. FIG. 7A shows a computer display 600 including relatively denseinformation 602 with various icons 604 for user interaction, forexample, via a mouse (not shown). The computer display of FIG. 7A may beprovided by the computer system when the user proximity context is thatthe user is relatively near the computer system (or the monitor thereof,for example).

When the user proximity context changes by the user moving to a locationrelatively far from the computer system (or the monitor thereof, forexample), the relatively dense information 602 may be filtered and/orenlarged to provide information that is usable/viewable from theincreased distance. As shown in FIG. 7B, for example, such a change inuser proximity context may result in a computer display 600′ includingfiltered information 602′, which is an enlarged portion of therelatively dense information 602 shown in FIG. 7A. The icons 604 andsome of the information 602 shown in the display 600 of FIG. 7A may notbe included, again, because the user may be at a location where the usercan use such icons 604 or information 602. Further, the appearance maybe altered, in addition to or instead of enlarging, for example, bychanging information/background to enhance uasability/viewability fromthe increased distance.

Other features of the display or the computer system may be controlledor otherwise may respond to changes in user proximity contexts and/oruser presence contexts. For example, screen brightness may be controlledbased on such user contexts to automatically adjust for changing viewingconditions for the user.

Also, for example, content may be hidden/viewable and/or available oractive applications may be enabled/disabled based upon user presencecontexts. Private or sensitive information may only be viewable when theuser presence context is that a single user is present, defining aprivate user presence context. Such information may be automaticallyhidden, for example, when a change in the user presence context isdetermined by detecting the presence of another person in a vicinity ofthe computer system (or the monitor thereof, for example). Closing orhiding windows based on such a change in user presence context to apublic user context may present the private or sensitive informationfrom being viewed by others, even if the user is unaware of the otherperson's presence or forgets that the information is viewable, privateor sensitive.

In general, it should be understood that any desired feature of acomputer system may be controlled or enabled to automatically respond tochanges in user contexts using a sensor-based user interface asdescribed herein. Thus, the foregoing descriptions should be understoodto be illustrative, and not exhaustive.

In particular, as discussed above, the sensor-based user interfacesystems and methods described herein may allow a user to control and/orto interact with a computer system remotely, without the use of a usermanipulatable input device or remote controller. In some embodiments,the sensor-based user interface systems and methods may be configured todetect a variety of user movements and/or gestures to determine varioususer gesture contexts. The various determined user gesture contexts mayhave corresponding control instructions for the computer system to causethe computer system to perform various operations, which may depend, forexample, on an application that is active on the computer system.

The user gesture contexts that are determined based on the detectedmovements and/ort gestures of the user may define an entire set of“multi-touch” display gestures, that is, a set of gestures analogous tothe multi-touch gestures developed in other technologies. For example,gestures analogous to those disclosed in the following U.S. patents andpublished U.S. patent applications may be employed, as appropriate ordesired, with the computer user interface systems and methods describedherein: U.S. Pat. Nos. 6,323,846, 6,570,557, 6,888,536, 7,030,861,7,339,580 and RE40,153; and U.S. patent publications nos. 2006/0026521,2006/0026535, 2006/0026536, 2007/0257890, 2008/0036743, 2008/0158167,2008/0158168, 2008/0158172, 2008/0204426, 2008/0211775, 2008/0211783,2008/0211784 and 2008/0211785. For example, movement of the user's handleft-to-right, right-to-left, up-to-down, and down-to-up may effect arespective scrolling control. Movement of a user's hand fromfar-to-close may effect a selecting control. Movement of the user'shands from left and right-to-middle or from up and down-to-middle mayeffect a zooming control. The opposite directional gesture may zoom out.

A simple example of pseudo-code for a “scroll left-to-right gesture” maybe as follows. A user interface system including three distance/depthsensors disposed on a computer monitor, for example, a left sensor (LS),a middle sensor (MS) and a right sensor (RS), may be configured todetect a user at distances up to a couple feet, for example, with anangular directionality of less than five degrees. The sensor processingof the system may be configured to produce a distance function DISTANCEwith values of NEAR, MIDRANGE and FAR from a set of distancemeasurements d and a presence function PRESENCE with values of YES andNO.

At any discrete time sampling point t=K:

USER_PRESENT (K) = if PRESENCE (LS, K) or PRESENCE (RS, K) or PRESENCE(MS, K) d (USER, K) = weighted_average (d (LS), d (MS), d (RS)) at timet = K DISTANCE (S, K) = F (d(S), K), where F is a trinity function and Sis LS, MS or RS USER_DISTANCE (K) = DISTANCE (d (USER, K)) ifUSER_PRESENT (K) then if AND ( DISTANCE (LS, K−3) = MIDRANGE andDISTANCE (LS, K−2) = MIDRANGE, PRESENCE (MS, K−3) = NO and PRESENCE (MS,K−2) = NO, PRESENCE (RS, K−3) = NO and PRESENCE (RS, K−2) = NO, ) andAND( PRESENCE (LS, K−1) = NO, PRESENCE (MS, K−1) = YES and DISTANCE (MS,K−1) = MIDRANGE, PRESENCE (RS, K−1) = NO ) and AND ( PRESENCE (LS, K) =NO, PRESENCE (MS, K) = NO and DISTANCE (MS, K−1) = MIDRANGE, PRESENCE(RS, K) = YES and DISTANCE (RS, K) = MIDRANGE ) then GESTURE =SCROLL_LEFT_TO_RIGHT endif endif

The control instructions provided to the computer system for each of theuser gesture contexts may be applied to an active (e.g., topmost)application, or to an entire desktop view, as appropriate or desired.

One example of a method of user interaction with a computer system via asensor-based user interface system is illustrated in FIG. 7. In theillustrated example, control may begin at S700 and continue to S710,where a baseline for user parameters and/or user contexts may bedetermined. For example, as discussed above, the sensor-based userinterface system may be trained to recognize a particular user'smovements. This may be done, for example, by having the user complete aseries of movements corresponding to specific user contexts so that thesensor-based user interface system may recognize such movements by theuse as intended to be the corresponding user context.

It should be understood that the operations of S710 are optional andneed not be included, for example, where the user gesture contexts arerelatively simple and/or limited. Also, once the sensor-based userinterface system has been “trained” for a particular user, thesensor-based user interface system may operate without such operationsbeing performed, for example, with only a user identification operationthat accesses user gesture context information stored based on thetraining. Alternatively or additionally, the “training” of thesensor-based user interface system may be performed and/or continueduring its normal use.

Normal use may be defined by S720 through S780 illustrated in FIG. 7,for example. As S720, gesture control may be initiated. This may occurautomatically, for example, after training is complete or upon startupof the sensor-based user interface system. Alternatively, this may bebased on a conventional user input, for example, where the sensor-baseduser interface system includes different modes of operation, forexample, for determining different user contexts and/or using differentsensors.

Next, at S730, a user movement or gesture (user parameter) may bedetected by the one or more sensors of the sensor-based user interfacesystem. The data or information from the sensor(s) may be processed atS740, such as described above. Based on the processing, a determinationmay be made at S750 as to whether a user gesture context is recognized.If not, control may return to S730 to continue to detect user movementsor gestures.

If a user gesture context is recognized at S750, control may proceed toS780 where a control instruction corresponding to the recognized gesturecontext may be obtained and output to the computer system. Thus, thecomputer system may implement an operation based on the controlinstruction corresponding to the recognized gesture context.

Next, at S770, a determination may be made as to whether gesture controlhas been ended. If not, control may return to S730 to continue to detectuser movements or gestures. If so, control may proceed to S780 and end.In some embodiments, a gesture context may have a corresponding controlinstruction that end gesture control. However, it should be understoodthat gesture control may be ended in any appropriate or desired manner.

The foregoing merely illustrates certain principles of the invention.Various modifications and alterations to the described embodiments willbe apparent to those skilled in the art in view of the teachings herein.It will thus be appreciated that those skilled in the art will be ableto devise numerous systems, arrangements and methods which, although notexplicitly shown or described herein, embody the principles disclosed inthis document and are thus within the spirit and scope of the presentinvention. From the above description and drawings, it will beunderstood by those of ordinary skill in the art that the particularembodiments shown and described are for purposes of illustration onlyand are not intended to limit the scope of the present invention.References to details of particular embodiments are not intended tolimit the scope of the invention.

1. A computer system comprising: at least one sensor, wherein the atleast one sensor is configured to gather information on a user within avicinity of the computer system; a processor, wherein the processor isconfigured to determine whether the user is within a predetermineddistance from the computer system based on the information; and adisplay screen, wherein the processor is configured to alter anappearance of information displayed on the display screen based onwhether the user is within the predetermined distance from the computersystem.
 2. The computer system defined in claim 1 wherein the at leastone sensor comprises wireless communications circuitry and wherein theat least one sensor is configured to detect the user's wirelesscommunication device with the wireless communications circuitry.
 3. Thecomputer system defined in claim 1 wherein the processor is configuredto enlarge the appearance of information displayed on the display screenin response to determining that the user is not within the predetermineddistance from the computer system.
 4. The computer system defined inclaim 1 wherein the processor is configured to remove at least some ofthe information from the display screen in response to determining thatthe user is not within the predetermined distance from the computersystem.
 5. The computer system defined in claim 1 wherein the processoris configured to adjust a brightness of the display-screen based onwhether the user is within the predetermined distance from the computersystem.
 6. The computer system defined in claim 1 further comprising aninput device, wherein the computer system is operable in a first controlmode and a second control mode, wherein the computer system isconfigured to receive user input via the input device in the firstcontrol mode, wherein the computer system is configured to receive userinput via gesture detection in the second control mode, and wherein theprocessor is configured to transfer the computer system from the firstcontrol mode to the second control mode in response to determining thatthe user is not within the predetermined distance from the computersystem.
 7. The computer system defined in claim 6 wherein the at leastone sensor comprises a plurality of sensors and wherein the plurality ofsensor configured to detect a gesture from the user in the secondcontrol mode.
 8. The computer system defined in claim 7 wherein theprocessor is configured to perform at least one operation based on thedetected gesture and wherein the at least one operation is selected fromthe group consisting of: a scrolling operation, a selecting operation,and a zooming operation.
 9. A computer system, comprising: at least onesensor, wherein the at least one sensor is configured to detect when asingle user is present and when multiple users are present; and aprocessor, wherein the processor is configured to place the computersystem in a private mode in response to the at least one sensordetecting that the single user is present and is configured to place thecomputer system in a public mode in response to the at least one sensordetecting that the multiple users are present.
 10. The computer systemdefined in claim 9 wherein private content is displayed on the displayscreen during the private mode.
 11. The computer system defined in claim10 wherein the private content is not displayed on the display screenduring the public mode.
 12. The computer system defined in claim 11wherein the processor is configured to disable an active application inresponse to the at least one sensor detecting that the multiple usersare present.
 13. The computer system defined in claim 9 wherein the atleast one sensor is configured to determine when no users are present,wherein the processor is configured to place the computer in a non-usemode in response to the at least one sensor determining that no usersare present, and wherein the non-use mode is selected from the groupconsisting of: a sleep mode, a hibernate mode, and a user logged outmode.
 14. A computer system, comprising: a display screen; at least onesensor configured to receive user input from a user in a first userinput mode and configured to gather proximity information from the user;an input device coupled to the display screen, wherein the input deviceis configured to receive user input from the user in a second user inputmode; and a processor configured to switch the computer system betweenthe first user input mode and the second user input mode based on theproximity information gathered by the at least one sensor.
 15. Thecomputer system defined in claim 14 wherein the input device comprises amouse.
 16. The computer system defined in claim 14 wherein the at leastone sensor is selected from group consisting of: a light sensor, avibration sensor, and an infrared sensor.
 17. The computer systemdefined in claim 14 wherein the at least one sensor comprises aplurality of sensors and wherein the plurality of sensors is configuredto detect gestures from the user in the first user input mode.
 18. Thecomputer system defined in claim 14 wherein the at least one sensor islocated in the display screen.
 19. The computer system defined in claim14 wherein the at least one sensor is located in the user input device.20. The computer system defined in claim 14 wherein the at least onesensor is separate from the display screen and is configured tocommunicate wirelessly with the processor.